Papers with self-attention block
DiffSkip: Differential Layer Skipping in Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing Large Language Models (LLMs) enforce uniform computation across all tokens. |
| Approach: | They propose to dynamically skip FFN blocks based on self-attention difference . they propose to use a lightweight router module to do the same . |
| Outcome: | The proposed model can skip FFN blocks even in token generation tasks. |
Understanding How Positional Encodings Work in Transformer Model (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have reported superiority of relative PEs in translation tasks. |
| Approach: | They analyze in which part of a transformer model PEs work and compare them using experiments . they find that relative PEs should be added only to query and key of attention mechanism . |
| Outcome: | The results show that relative and absolute PEs work in a transformer model, and should be added to the query and key of an attention mechanism, not to the value. |